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DETAILED ACTION 

This office action is responsive to the Request for Continued Examination (RCE) 
filed August 6, 2008. Claims 10, 18, 27 and 28 are currently amended. Therefore 
claims 10, 11, 15, 17, 18 and 20-28 remain pending in this application. 



Claim Objections 

1 . Claims 10, 27 and 28 are objected to because of the following informalities: the 
preamble of claims 10, 27 and 28 should be amended to recite a "computer- 
implemented method" to show that the method is performed by a computer. 
Appropriate correction is required. 



Claim Rejections - 35 USC § 112 

2. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

3. Claims 10, 27 and 28 are rejected under 35 U.S.C. 112, second paragraph, as 
being indefinite for failing to particularly point out and distinctly claim the subject matter 
which applicant regards as the invention. 

4. Claim 10 recites the limitation "the pixels" in line 6 of the claim. There is 
insufficient antecedent basis for this limitation in the claim. 

5. Claim 27 recites the limitation "the pixels" in line 5 of the claim. There is 
insufficient antecedent basis for this limitation in the claim. 
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6. Claim 27 recites the limitation "the additional context information" in line 19 of the 
claim. There is insufficient antecedent basis for this limitation in the claim. 

7. Claim 28 recites the limitation "the pixels" in line 5 of the claim. There is 
insufficient antecedent basis for this limitation in the claim. 

Claim Rejections - 35 USC § 103 

8. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

9. Claims 10, 20-24 are rejected under 35 U.S.C. 103(a) (current application filing 
date 1/26/2004) as being unpatentable over Denoue et al. (US 2004/0119762 A1, filing 
date 12/24/2002) ('Denoue') in view of Dawe et al. (US 7,042,594 B1 , filing date 
3/7/2000) ('Dawe'). 

With respect to claim 10, Denoue teaches: 

receiving a path drawn by a user via a stylus as input (paragraphs 54 and 86), 
the path defining boundaries of a selected region of a display (paragraph 59), one or 
more graphical elements being displayed in the selected on-screen region (Figure 6, 
paragraph 67); 

capturing the pixels displayed within the selected on-screen region (paragraphs 
67 and 86-87), and storing the captured image pixels such that the stored data is 
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representative of only those pixels of the display within the selected on screen region 
(paragraph 89); and 

obtaining context information for the one or more graphical elements by 
automatically applying text recognition (paragraph 79) to an annotation drawn by the 
user on the display via the stylus {i.e. freeform inks 532, 534, 536 and freeform notes) 
(paragraphs 57 and 75) in proximity of the selected on-screen region (OCR may be 
applied to captured content and captured content may include freeform inks, as seen in 
figs. 2-5), 

wherein context information is stored in association with the captured data 
(paragraphs 75-76). 

Denoue does not explicitly teach storing captured data in an image file; storing 
the results of text recognition as context information; or automatically storing context 
information in association with the image file. 

Dawe teaches a system and method for saving handwriting as an annotation in a 
scanned document (see abstract), in which he teaches: 

capturing image pixels and storing captured image pixels in an image file (col. 3 
lines 60-63, col. 7 line 61- col. 8 line 4); 

applying text recognition to annotations and storing the results of text recognition 
as context information (col. 5 lines 52-60, col. 7 lines 6-20 and 61-65); and 

automatically storing context information in association with the image file (col. 7 
lines 57-65). 
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It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Denoue by the teaching of Dawe because 
storing captured data in an image file; storing the results of text recognition as context 
information; or automatically storing context information in association with the image 
file would enable handwriting to be saved as an annotation, and thus reduce clutter in a 
document presented to a user, while maintaining the information conveyed by the 
handwriting for presentation to the user if desired (Dawe, abstract). 

With respect to claim 20, Denoue as modified teaches wherein the context 
information is stored in such a manner as to be accessible to a user for performing at 
least one of the following: 

searching for said image file; 

displaying the context information simultaneously with the captured image pixels 
(Denoue, paragraphs 75-76; Dawe, col. 8 lines 1-16), and 

navigating a network to a source of the captured image pixels. 

With respect to claim 21 , Denoue as modified teaches wherein the one or more 
graphical elements comprises a first set of one or more textual characters, the method 
further comprising: obtaining the context information as text data obtained by performing 
text recognition on at least one of: the first set of one or more textual characters, and a 
second set of textual characters displayed in proximity with the first set (Dawe: col. 7 
lines 6-20 and lines 61-65; Denoue, paragraph 72). 



Application/Control Number: 10/766,319 



Art Unit: 2164 



Page 6 



With respect to claim 22, Denoue as modified teaches wherein the selected on- 
screen region is part of displayed textual region, and the graphical elements comprise a 
first set of one or more textual characters displayed in the textual region, the method 
further comprising: obtaining the context information based on a second set of one or 
more textual characters displayed in the textual region (Dawe, Fig. 4, col. 7 lines 6-20 
and lines 61-65; Denoue, paragraph 72) {The words "job" or "awareness" may be 
considered the second set of textual characters). 

With respect to claim 23, Denoue as modified teaches wherein the step a) 
receives the user input based on movement of a stylus across the display (Denoue, 
paragraph 54; Dawe, col. 5 lines 28-30). 

With respect to claim 24, Denoue as modified teaches: 

digitizing movements of a stylus across the display in order to receive the 

annotation (Denoue, paragraphs 55, 57 and 75); and 

obtaining the context information based on the received annotation (Denoue, 

paragraph 79). 

10. Claim 11, 25 and 26 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), as applied to claims 10 
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and 20-24 above, and further in view of Oppermann et al. (US 6,334,157 B1, filing date 
3/11/1997) ('Oppermann'). 

With respect to claim 1 1 , Denoue as modified teaches claim 10, including a 
selected on-screen region. 

Denoue as modified does not teach determining a window associated with the 
selected on-screen region; retrieving an application interface having a uniform resource 
identifier (URI) property from the determined window or parent window of the 
determined window; or obtaining the URI property as the context information. 

Oppermann teaches programmatically providing direct access to user interface 
elements of an application program (see abstract), in which he teaches: 

selecting user interface elements, such as text (col. 8 lines 43 and 49-51) and 
determining a window associated with the selected elements (col. 26 lines 37-40); 

retrieving an application interface having a uniform resource identifier (URI) 
property from the determined window or parent window of the determined window (col. 
25 lines 59-62, col. 28 lines 33-39); and 

obtaining the URI property as the context information (col. 1 1 lines 1-9, 55-60, 
col. 12 lines 55-60, col. 13 lines 51-60). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Oppermann 
because determining a window, retrieving an application interface having a uniform 
resource identifier (URI) property from the determined window or parent window of the 
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determined window would enable accessibility aids the ability to access and manipulate 
user interface elements of any application program without having prior knowledge of 
the application program or its interface (Oppermann, column 4 lines 27-30). 

With respect to claim 25, Denoue as modified teaches wherein the selected on- 
screen region includes at least a portion of a displayed web page or document (Dawe, 
Figure 4, col. 4 lines 38-42), and the step d) further comprises: using an application 
programming interface (API) to query the application for the context information 
(Oppermann, column 7 lines 36-38). 

With respect to claim 26, Denoue as modified teaches wherein the step d) further 
comprises obtaining a uniform resource identifier (URI) of the web page or document as 
the context information (Oppermann, col. 11 lines 1-9, 55-60, col. 12 lines 55-60, col. 13 
lines 51-60), the URI being obtained as a result of the query using the API (Oppermann, 
column 7 lines 36-38). 

11. Claims 15 and 18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), as applied to claims 10 
and 20-24 above, and further in view of Browne et al. (US 2004/0135815 A1 , filing date 
12/15/2003) ('Browne'). 
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With respect to claim 15, Denoue as modified teaches claim 10. 

Denoue as modified does not teach creating and storing a linking structure as the 
association between the image file and the context information. 

Browne teaches a method and apparatus for image metadata entry (see 
abstract), in which he teaches creating and storing a linking structure as the association 
between the image file and the context information (Figure 12, paragraph 136). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Browne 
because teach creating and storing a linking structure as the association between the 
image file and the context information would enable an easy and efficient method of 
classifying and storing digital images (Browne, paragraph 14). 

With respect to claim 18, Denoue as modified teaches wherein the linking 
structure includes at least one pointer pointing to the stored image file or the stored 
content information (Browne, paragraph 136). 

12. Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over Denoue 
et al. (US 2004/01 1 9762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe et al 
(US 7,042,594 B1, filing date 3/7/2000) ('Dawe') and Browne et al. (US 2004/0135815 
A1 , filing date 12/15/2003) ('Browne'), as applied to claims 15 and 18 above, and further 
in view of Newman (US 2003/0101 156 A1 , filing date 11/26/2001). 
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With respect to claim 17, Denoue as modified teaches claim 15. 

Denoue as modified does not teach wherein the linking structure is incorporated 
in a file separate from the stored image file and the stored content information. 

Newman teaches database systems and methods (see abstract), in which he 
teaches wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information (paragraph 16). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Newman 
because wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information would enable additional information about 
image files, such as the origination device, person who created the file, and data/time 
the file was created, to be transmitted and stored along with the image files (Newman, 
paragraph 16). 

13. Claims 27 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), as applied to claims 10 and 
20-24 above, and further in view of Hertzfeld et al. (US 2002/00761 09 A1 , filing date 
1/25/1999) ('Hertzfeld') 



With respect to claim 27, Denoue teaches: 
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receiving a path drawn on the display by a user via a stylus (paragraphs 54 and 
86), the drawn path defining the boundaries of a selected on-screen region of the 
display (Figure 6, paragraphs 59 and 67); 

capturing the pixels displayed within the boundaries of the on-screen region 
(paragraphs 67 and 86-87); and 

storing the captured image pixels such that the stored data is representative of 
only those pixels of the display within the selected on screen region (paragraph 89); 
and 

extracting a character or word from textual data as context information 
(paragraph 79), 

wherein context information, and the additional context information, if any, is 
stored in association with the captured data (paragraphs 75-76). 

The limitation of storing additional context information, if any, is optional because 
it only occurs if there is additional context information. Furthermore, as explained 
above, "the additional context information" refers to any context information. 

Denoue does not teach: storing captured data in an image file; automatically 
determining whether the content displayed within the on-screen region includes textual 
data; when the displayed content of the on-screen region is determined to include 
textual data, automatically extracting a character or word from the textual data as 
context information; or storing the context information and the additional context 
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information, if any, in association with the image file, such that the context information is 
accessible when viewing the image file. 

Dawe teaches a system and method for saving handwriting as an annotation in a 
scanned document (see abstract), in which he teaches: 

capturing image pixels and storing captured image pixels in an image file (col. 3 
lines 60-63, col. 7 line 61- col. 8 line 4); 

automatically determining whether the content displayed within the on-screen 
region includes textual data (col. 6 lines 52-53, col. 7 lines 11-14); 

when the displayed content of the on-screen region is determined to include 
textual data, automatically extracting a character or word from the textual data as 
context information (col. 7 lines 6-8 and 15-20); and 

storing the context information and the additional context information, if any, in 
association with the image file, such that the context information is accessible when 
viewing the image file (col. 7 lines 57-60, col. 8 lines 1-16). 
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It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Denoue by the teaching of Dawe to enable 
handwriting to be saved as an annotation, and thus reduce clutter in a document 
presented to a user, while maintaining the information conveyed by the handwriting for 
presentation to the user if desired (Dawe, abstract). 

Further regarding claim 27, Denoue in view of Dawe does not teach 
automatically determining whether the displayed content of the on-screen region 
includes underlying data comprising at least one of: an executable object, a file, and a 
link to remote content; or when the displayed content of the on-screen region is 
determined to include underlying data, automatically extracting a property of the 
underlying data as context information, the property comprising at least one of: a file 
name, a file identifier, a uniform resource locator (URL), a uniform resource identifier 
(URI), a folder name, and meta-data. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches: 

automatically determining whether the displayed content of the on-screen region 
includes underlying data comprising at least one of: an executable object, a file, and a 
link to remote content (step 608 in Figure 6, paragraph 38); and 

when the displayed content of the on-screen region is determined to include 
underlying data, automatically extracting a property of the underlying data as context 
information, the property comprising at least one of: a file name, a file identifier, a 
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uniform resource locator (URL), a uniform resource identifier (URI), a folder name, and 
meta-data (step 616 in Figure 6, paragraph 38). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Hertzfeld 
because automatically determining whether the displayed content of the on-screen 
region includes underlying data comprising at least one of: an executable object, a file, 
and a link to remote content; or if the displayed content of the on-screen region is 
determined to include underlying data, automatically extracting a property of the 
underlying data as context information, the property comprising at least one of: a file 
name, a file identifier, a uniform resource locator (URL), a uniform resource identifier 
(URI), a folder name, and meta-data would enable recognition of predefined types of 
text and predefined actions to be performed based on the types of text (Hertzfeld, 
abstract). 

The limitations "automatically extracting a character or word from the textual data 
as context information" and "automatically extracting a property of the underlying data 
as context information, the property comprising at least one of: a file name, a file 
identifier, a uniform resource locator (URL), a uniform resource identifier (URI), a folder 
name, and meta-data" are conditional statements, and thus optionally patentable. The 
limitations only occur if/when another action occurs, and thus do not limit the claim. 

14. Claims 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Dawe 
et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Beauchamp et al. (US 
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2008/0046837 A1 , filing date 3/17/2003) ('Beauchamp'), and further in view of Denoue 
et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue'). 

With respect to claim 28, Dawe teaches: 

receiving a path drawn on the display by a user via a stylus (col. 5 lines 28-30; 
col. 6 lines 48-50), the drawn path defining the boundaries of a selected on-screen 
region of the display (col. 6 lines 42-51); 

capturing the pixels within the boundaries of the on-screen region (col. 3 lines 60- 
63, col. 7 lines 40-42 and lines 61-65, col. 8 lines 1-3); 

storing the captured pixels as an image file (col. 3 lines 60-63, col. 7 lines 40-42 
and lines 61-65, col. 8 lines 1-3); 

performing text recognition on an annotation to produce recognized text of the 
annotation as context information (col. 5 lines 36-43, col. 7 lines 6-20, col. 8 lines 19- 
29); 

automatically determining that the content displayed within the on-screen region 
includes at least one of textual data and other underlying data comprising at least one 
of an executable object, a file, and a link to remote content (col. 6 lines 52-53, col. 7 
lines 11-20); 

automatically extracting as additional context information at least one of: 
a character or word from the textual data as context information (col. 7 lines 6-20 
and 57-65, col. 8 lines 19-26); and 
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a property of underlying data determined to be included in the on-screen region, 
the property comprising at least one of: a file name, a file identifier, a uniform resource 
locator (URL), a uniform resource identifier (URI), a folder name, and meta-data; and 

storing the context information and the additional context information in 
association with the image file, such that the context information is accessible when 
viewing the image file (col. 7 lines 57-60, col. 8 lines 1-16). 

Dawe does not teach receiving an annotation drawn on the display by the user 
via the stylus; or performing text recognition of the annotation (drawn on the display) to 
produce recognized text of the annotation as context information. 

Beauchamp teaches a transparent windows method and apparatus (see 
abstract), in which he teaches: 

receiving an annotation drawn on the display by the user via the stylus 
(paragraph 6); and 

performing text recognition of the annotation (drawn on the display) to produce 
recognized text of the annotation as context information (paragraph 6). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Dawe by the teaching of Beauchamp 
because receiving an annotation drawn on the display by the user via the stylus; or 
performing text recognition of the annotation (drawn on the display) to produce 
recognized text of the annotation as context information would enable optimization of 
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pen-based annotations on a window of a software application (Beauchamp, paragraph 
8). 

Further regarding claim 28, Dawe in view of Beauchamp does not teach storing 
captured pixels such that the image file is representative of only those pixels of the 
display within the on-screen region. 

Denoue teaches systems and methods for freeform pasting (see abstract), in 
which he teaches capturing content, and storing captured content (pixels) such that the 
image file is representative of only those pixels of the display within the on-screen 
region (paragraph 89). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Dawe by the teaching of Denoue 
because storing captured content would enable the user or other users to ability to 
reselect content in a simplified manner (Denoue, abstract, paragraph 89). 

Response to Arguments 

1 5. Applicant's arguments with respect to claims 10, 11, 15, 17, 18 and 20-28 have 
been considered but are moot in view of the new ground(s) of rejection. 

16. Regarding claim 27, Applicant argues that Hertzfeld does not teach determining 
whether the displayed content of the on-screen region includes underlying data 
comprising at least one of: an executable object, a file, and a link to remote content. 
Examiner disagrees. Hertzfeld teaches that an interpreter determines if selected text 
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includes a World Wide Web network address (paragraph 38). It is clear that a World 
Wide Web address may be considered a link to remote content or a file. Furthermore, 
Hertzfeld teaches that when it is determined that the selected text includes a web 
address, the system connects to the Internet to display the corresponding web page. 
Therefore, it is clear that Hertzfeld determines whether the content includes underlying 
data such as a link to remote content or a file. 

1 7. The Examiner would also like to point out that changing the "if claim language to 
"when" does not change the conditional status of limitations including such claim 
language. The claim language "when" is also considered optional because it does not 
require the step to actually occur. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Alicia M. Lewis whose telephone number is 571-272- 
5599. The examiner can normally be reached on Monday - Friday, 9 - 6:30, alternate 
Friday off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Charles Rones can be reached on 571-272-4085. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/A. M. U 

Examiner, Art Unit 2164 
/Charles Rones/ 

Supervisory Patent Examiner, Art Unit 2164 



